prep_wake_samples --clean: clear a re-recorded take's orphan clips (#8) - #85
Merged
Conversation
…ips (#8) Re-recording a take with fewer utterances and re-running prep into a populated --output dir left the previous run's higher-numbered clips on disk. Training and eval glob every clip in the directory, so those orphans were silently folded into the next run's data while process() reported only the count it had just written. prep already detected the leftovers and warned; --clean now removes them in the same run so the reported count matches what is on disk. Deletion is scoped to the current run's own stems -- a "<stem>_NNN.wav" clip whose stem this run wrote -- so a source recording, an unrelated take's clips, or hand-curated audio in the directory is named but never auto-deleted. A stray --output can't wipe files prep did not create, and without --clean the warn-and-name behavior is unchanged. The stem match is a greedy "(.+)_\d{3,}\.wav" so an orphan pairs back to its disambiguated collider stem (computah-1_002.wav -> computah-1, not computah) and the >999-segment case (:03d is a minimum width, so index 1000 is take_1000.wav) still parses. Tests cover the re-run-with-fewer-clips case with and without --clean, that --clean spares files this run did not record, and disambiguated-stem orphan cleanup. Orphans of a stem the current run does not write -- a dropped input, or a collider whose disambiguated stem changed between runs -- stay warn-only, since prep can't tell them from data the user curated on purpose. A manifest would let --clean reach those safely; filed as #84. Refs #8 wake-20260723T1005-20d30f
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 990795d801
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The cloud review caught a data-loss hole in the --clean stem-scoping: run_stems was built from the input list (`stems`), so a take that produced zero clips this run -- a silent or bad re-recording, or thresholds that dropped every segment -- still had its stem in run_stems, and --clean then deleted that stem's prior good clips while writing no replacement. The narrow "only a re-recorded take's own orphans" guarantee has to key off what the run actually wrote, not what it tried to record. Derive run_stems from `written` (each written path is a `<stem>_NNN.wav`, so its stem reads back through _clip_stem). A take that wrote nothing contributes no stem, so its earlier clips linger and warn instead of being wiped, and the only copy stays safe. test_clean_spares_prior_clips_when_rerun_writes_nothing reproduces it: a good 3-clip run, then a rerun whose --min-dur drops every segment; with --clean the prior clips must survive. It fails on the old input-stem scoping and passes on the written-stem scoping. Refs #8 wake-20260723T1005-20d30f
This was referenced Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a
--cleanflag toprep_wake_samples.py. Re-recording a take with fewerutterances and re-running prep into a populated
--outputdir left the previousrun's higher-numbered clips on disk. Training and eval glob every clip in the
directory, so those orphans were silently folded into the next run's data while
process()reported only the count it had just written.--cleanremoves them inthe same run so the reported count matches what is on disk.
Why it's safe
Deletion is scoped to the current run's own stems: a
<stem>_NNN.wavclip whosestem this run wrote. A source recording, an unrelated take's clips, or hand-curated
audio in the directory is named in the output but never auto-deleted, so a stray
--outputcan't wipe files prep did not create. Without--cleanthe existingwarn-and-name behavior is unchanged (deletion is opt-in).
The stem match is a greedy
(.+)_\d{3,}\.wav, so an orphan pairs back to itsdisambiguated collider stem (
computah-1_002.wav->computah-1, notcomputah)and the >999-segment case (
:03dis a minimum width, so index 1000 istake_1000.wav) still parses.Tests
test_prep_wake_samples.pygains coverage for the re-run-with-fewer-clips casewith and without
--clean, that--cleanspares files this run did not record,and disambiguated-stem orphan cleanup. Full suite: 31/31 checks,
ruff checkandruff format --checkclean.Known limit (deferred)
Orphans of a stem the current run does not write -- a dropped input, or a collider
whose disambiguated stem changed between runs -- stay warn-only, since prep can't
tell them from data the user curated on purpose. A manifest would let
--cleanreach those safely: filed as #84.
Refs #8
wake-20260723T1005-20d30f